Fix CLI register path dedup and local timm model export (#2417, #2423) - #2478
Open
Zhussup wants to merge 1 commit into
Open
Fix CLI register path dedup and local timm model export (#2417, #2423)#2478Zhussup wants to merge 1 commit into
Zhussup wants to merge 1 commit into
Conversation
- Deduplicate optimum.commands.register search locations by the resolved physical path so symlinked lib64 -> lib directories on RHEL no longer register subcommands twice or emit spurious __init__.py warnings (huggingface#2417). - Load local timm models with the `local-dir:` source prefix instead of `hf_hub:`, which timm rejects for local directory paths (huggingface#2423). - Add a "Where does my contribution belong?" section to CONTRIBUTING.md mapping features to their subpackage repositories. - Add regression tests for both fixes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes two issues addressable in the core
optimumrepository and updates the contributing guide to reflect the current split into subpackages.#2417 — sub-commands fail on RHEL (lib64 → lib symlink)
load_optimum_namespace_cli_commandsdeduplicatedsubmodule_search_locationswith a plainset()over the raw strings. On systems wherelib64is a symlink tolib(e.g. RHEL), this returns two entries pointing to the same physical directory, so subpackage commands were registered twice and spurious__init__.pywarnings were emitted.Fix: deduplicate by the resolved physical path (
Path.resolve()) and guarditerdir()with atry/except OSError.#2423 — cannot export local timm models
The timm loading branch always passed
hf_hub:{model_name_or_path}totimm.create_model. For a local directory, timm interprets thehf-hub/hf_hubsource prefix as a Hub repo id and fails to download it. timm does not accept a bare local path — thelocal-dir:source prefix is required for local folders.Fix: select the source prefix based on whether the argument is a local directory:
local-dir:{path}hf_hub:{path}(unchanged)CONTRIBUTING.md
Added a "Where does my contribution belong?" section mapping features/backends to their subpackage repositories (optimum-onnx, optimum-intel, optimum-habana, optimum-neuron, optimum-amd, optimum-nvidia, optimum-quanto, optimum-furiosa, optimum-executorch), and clarifying that this repository holds the shared core (CLI,
optimum/exporters/tasks.py, utils).Tests
tests/cli/test_cli.py::TestCLI::test_load_namespace_cli_commands_dedup_symlink— regression test for Cannot run sub-commands on RHL becauseoptimummishandles the lib64 symlink to lib #2417 using an isolated temp directory plus a symlink duplicate and a namespaceModuleSpec.tests/exporters/common/test_timm_loading.py— verifies thelocal-dir:prefix is used for a local path andhf_hub:is preserved for a Hub repo id (mockstimm.create_model, no network/weights).Both pass with
transformers 4.57.6andtransformers 5.15.0.Notes from triage (no code change needed in this repo)
AutoModelFor*loaders referenced intasks.pyexist on transformers 5.15, the version gate attasks.py:156picksAutoModelForImageTextToText, andsetup.pyhas no upper cap on transformers. Remaining blockers are in the subpackages (e.g. optimum-onnx pins transformers < 5).qwen3_asrONNX config, which lives in optimum-onnx.optimum[onnx]+onnxruntime(openai/whisper-tinyandsshleifer/tiny-gpt2export cleanly); these belong to the optimum-onnx export layer.